Preprocessing QC statistics ¶

October 2024¶

In [17]:
import os
NOVA_HOME = ''
NOVA_DATA_HOME = ''
LOGS_PATH = os.path.join(NOVA_DATA_HOME, "outputs/preprocessing/spd/logs/preprocessing_Dec2023")
PLOT_PATH = os.path.join(NOVA_HOME, 'src', 'preprocessing', 'notebooks','figures','Neurons')
os.chdir(NOVA_HOME)
import pandas as pd
import contextlib
import io
from IPython.display import display, Javascript

from tools.preprocessing_tools.qc_reports.qc_utils import log_files_qc, run_validate_folder_structure, display_diff, sample_and_calc_variance, \
                                                show_site_survival_dapi_brenner, show_site_survival_dapi_cellpose, \
                                                show_site_survival_dapi_tiling, show_site_survival_target_brenner, \
                                                calc_total_sums, plot_filtering_heatmap, show_total_sum_tables, \
                                                plot_cell_count, plot_catplot, plot_hm_combine_batches, plot_hm, \
                                                run_calc_hist_new
                                                
from tools.preprocessing_tools.qc_reports.qc_config import panels, markers, marker_info, cell_lines, cell_lines_to_cond,\
                                    cell_lines_for_disp, reps, line_colors, lines_order, custom_palette,\
                                    expected_dapi_raw
%load_ext autoreload
%autoreload 2

The autoreload extension is already loaded. To reload it, use:
  %reload_ext autoreload
In [3]:
# choose batches
batches = [f'batch{i}' for i in range(4,10)]
In [21]:
df = log_files_qc(LOGS_PATH, batches)

df_dapi = df[df.marker=='DAPI']
df_target = df[df.marker!='DAPI']

reading logs of batch4_original_with_brenner
reading logs of batch9_original_with_brenner
reading logs of batch6_original_with_brenner
reading logs of batch7_original_with_brenner
reading logs of batch5_original_with_brenner
reading logs of batch8_original_with_brenner

Total of 10 files were read.
Before dup handeling  (360346, 22)
After duplication removal #1: (354100, 23)
After duplication removal #2: (350313, 23)

Actual Files Validation¶

Raw Files Validation¶

  1. How many site tiff files do we have in each folder?
  2. Are all existing files valid? (tif, at least 2049kB, not corrupetd)
In [5]:
root_directory_raw = os.path.join(NOVA_DATA_HOME, 'input', 'images', 'raw', 'SpinningDisk')

batches_raw = [batch.replace("_16bit_no_downsample","") for batch in batches]
raws = run_validate_folder_structure(root_directory_raw, False, panels, markers,PLOT_PATH,marker_info,
                                    cell_lines_to_cond, reps, cell_lines_for_disp, expected_dapi_raw,
                                     batches=batches_raw)

batch4
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
batch5
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
batch6
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
batch7
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
batch8
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
batch9
Folder structure is valid.
No bad files are found.
Total Sites:  64800

========
====================

Processed Files Validation¶

  1. How many site npy files do we have in each folder? -> How many sites survived the pre-processing?
  2. Are all existing files valid? (at least 100kB, npy not corrupted)
In [7]:
root_directory_proc = os.path.join(NOVA_DATA_HOME, 'input', 'images', 'processed', 'spd2',
                              'SpinningDisk')
procs = run_validate_folder_structure(root_directory_proc, True, panels, markers,PLOT_PATH,marker_info,
                                    cell_lines_to_cond, reps, cell_lines_for_disp, expected_dapi_raw,
                                     batches=batches)

batch4
Folder structure is invalid. Missing 1 paths:
/home/labs/hornsteinlab/Collaboration/MOmaps/input/images/processed/spd2/SpinningDisk/batch4/SCNA
No bad files are found.
Total Sites:  44040

========
batch5
Folder structure is invalid. Missing 1 paths:
/home/labs/hornsteinlab/Collaboration/MOmaps/input/images/processed/spd2/SpinningDisk/batch5/SCNA
No bad files are found.
Total Sites:  47883

========
batch6
Folder structure is valid.
No bad files are found.
Total Sites:  62299

========
batch7
Folder structure is valid.
No bad files are found.
Total Sites:  62761

========
batch8
Folder structure is valid.
No bad files are found.
Total Sites:  62235

========
batch9
Folder structure is valid.
No bad files are found.
Total Sites:  62415

========
====================

Difference between Raw and Processed¶

In [8]:
display_diff(batches, raws, procs, PLOT_PATH)

batch4

========
batch5

========
batch6

========
batch7

========
batch8

========
batch9

========

Variance in each batch (of processed files)¶

In [8]:
for batch in batches:
    with contextlib.redirect_stdout(io.StringIO()):
        var = sample_and_calc_variance(root_directory_proc, batch, 
                                       sample_size_per_markers=200, num_markers=26)
    print(f'{batch} var: ',var)

batch4 var:  0.014266254500867515
batch5 var:  0.015126299969755048
batch6 var:  0.016459885038877964
batch7 var:  0.01608249615403339
batch8 var:  0.01643550375364174
batch9 var:  0.016846129673614187

Preprocessing Filtering qc¶

By order of filtering

1. % site survival after Brenner on DAPI channel¶

Percentage out of the total sites

In [22]:
dapi_filter_by_brenner = show_site_survival_dapi_brenner(df_dapi,batches, line_colors, panels, reps)

2. % Site survival after Cellpose¶

Percentage out of the sites that passed the previous filter. In parenthesis are absolute values.

A site will be filtered out if Cellpose found 0 cells in it.

In [23]:
dapi_filter_by_cellpose = show_site_survival_dapi_cellpose(df_dapi, batches, dapi_filter_by_brenner, line_colors, panels, reps)

3. % Site survival by tiling¶

Percentage out of the sites that passed the previous filter. In parenthesis are absolute values.

A site will be filtered out if after tiling, no tile is containing at least one whole cell that Cellpose detected.

In [24]:
dapi_filter_by_tiling=show_site_survival_dapi_tiling(df_dapi, batches, dapi_filter_by_cellpose, line_colors, panels, reps)

4. % Site survival after Brenner on target channel¶

Percentage out of the sites that passed the previous filter. In parenthesis are absolute values (if different than the percentages).

In [25]:
show_site_survival_target_brenner(df_dapi, df_target, dapi_filter_by_tiling, markers)

Statistics About the Processed Files¶

In [26]:
stats = ['n_valid_tiles','site_whole_cells_counts_sum','site_cell_count','site_cell_count_sum']
total_sum = calc_total_sums(df_target, df_dapi, stats, markers)

Total tiles¶

In [27]:
total_sum[~total_sum.marker.str.contains('TIA', regex=True)].n_valid_tiles.sum()
Out[27]:
2316750
In [28]:
## Total tiles in wt lines
total_sum[(~total_sum.marker.str.contains('TIA', regex=True))&
         (total_sum.cell_line_cond.isin(['WT stress', 'WT Untreated']))].n_valid_tiles.sum()
Out[28]:
627301
In [29]:
## Total tiles in untreated lines
total_sum[(~total_sum.marker.str.contains('TIA', regex=True))&
         ((~total_sum.cell_line_cond.str.contains('WT')) | (total_sum.cell_line_cond=='WT Untreated'))].n_valid_tiles.sum()
Out[29]:
2010906

Total whole nuclei in tiles¶

In [30]:
total_sum[total_sum.marker =='DAPI'].site_whole_cells_counts_sum.sum()
Out[30]:
791561.0

Total nuclei in sites¶

In [31]:
total_sum[total_sum.marker =='DAPI'].site_cell_count.sum()
Out[31]:
1960072.0

Total tiles in training batches¶

In [32]:
total_sum[(total_sum.batch.str.contains('7|8', regex=True))
          & (~total_sum.marker.str.contains('TIA1', regex=True))].n_valid_tiles.sum()
Out[32]:
820972

Training: Total whole nuclei in tiles¶

In [33]:
total_sum[(total_sum.batch.str.contains('7|8', regex=True)) 
          & (total_sum.marker =='DAPI')].site_whole_cells_counts_sum.sum()
Out[33]:
274483.0
In [34]:
show_total_sum_tables(total_sum)
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch4
count 622.000000 622.000000 622.000000 622.000000
mean 477.554662 4.775547 518.152733 1200.618971
std 349.809784 3.498098 386.319223 907.862195
min 0.000000 0.000000 0.000000 0.000000
25% 140.500000 1.405000 144.000000 347.000000
50% 451.000000 4.510000 483.000000 1135.000000
75% 814.000000 8.140000 888.000000 2039.000000
max 1158.000000 11.580000 1294.000000 3027.000000
sum 297039.000000 NaN 322291.000000 746785.000000
expected_count 450.000000 450.000000 450.000000 450.000000
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch5
count 638.000000 638.000000 638.000000 638.000000
mean 528.673981 5.286740 575.824451 1340.454545
std 351.274384 3.512744 387.718429 921.444416
min 1.000000 0.010000 1.000000 1.000000
25% 191.500000 1.915000 207.750000 446.500000
50% 554.000000 5.540000 596.000000 1441.500000
75% 825.000000 8.250000 900.000000 2091.750000
max 1157.000000 11.570000 1281.000000 2988.000000
sum 337294.000000 NaN 367376.000000 855210.000000
expected_count 450.000000 450.000000 450.000000 450.000000
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch6
count 645.000000 645.000000 645.000000 6.450000e+02
mean 695.674419 6.956744 741.423256 1.947064e+03
std 178.072058 1.780721 193.933512 5.275288e+02
min 34.000000 0.340000 36.000000 8.400000e+01
25% 577.000000 5.770000 612.000000 1.569000e+03
50% 705.000000 7.050000 755.000000 1.976000e+03
75% 814.000000 8.140000 863.000000 2.324000e+03
max 1095.000000 10.950000 1189.000000 2.996000e+03
sum 448710.000000 NaN 478218.000000 1.255856e+06
expected_count 450.000000 450.000000 450.000000 4.500000e+02
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch7
count 648.000000 648.000000 648.000000 6.480000e+02
mean 668.841049 6.688410 712.540123 1.735125e+03
std 185.421616 1.854216 202.016806 5.084607e+02
min 129.000000 1.290000 138.000000 3.230000e+02
25% 536.000000 5.360000 567.000000 1.350750e+03
50% 668.000000 6.680000 702.000000 1.758000e+03
75% 804.000000 8.040000 855.000000 2.117500e+03
max 1178.000000 11.780000 1261.000000 3.164000e+03
sum 433409.000000 NaN 461726.000000 1.124361e+06
expected_count 450.000000 450.000000 450.000000 4.500000e+02
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch8
count 647.000000 647.000000 647.000000 6.470000e+02
mean 629.069552 6.290696 667.505410 1.677734e+03
std 173.432002 1.734320 187.446288 4.999782e+02
min 44.000000 0.440000 46.000000 9.200000e+01
25% 517.000000 5.170000 545.000000 1.348000e+03
50% 620.000000 6.200000 659.000000 1.652000e+03
75% 743.000000 7.430000 801.000000 2.060000e+03
max 1086.000000 10.860000 1189.000000 2.996000e+03
sum 407008.000000 NaN 431876.000000 1.085494e+06
expected_count 450.000000 450.000000 450.000000 4.500000e+02
n_valid_tiles % valid tiles site_whole_cells_counts_sum site_cell_count
batch9
count 645.000000 645.000000 645.000000 6.450000e+02
mean 703.877519 7.038775 751.767442 1.935143e+03
std 202.101326 2.021013 219.785183 5.705543e+02
min 92.000000 0.920000 97.000000 2.240000e+02
25% 557.000000 5.570000 593.000000 1.504000e+03
50% 717.000000 7.170000 762.000000 2.035000e+03
75% 859.000000 8.590000 924.000000 2.359000e+03
max 1205.000000 12.050000 1323.000000 3.215000e+03
sum 454001.000000 NaN 484890.000000 1.248167e+06
expected_count 450.000000 450.000000 450.000000 4.500000e+02
n valid tiles % valid tiles site_whole_cells_counts_sum site_cell_count
All batches
count 3.845000e+03 3845.000000 3.845000e+03 3.845000e+03
mean 6.183254e+02 6.183254 6.622567e+02 1.642620e+03
std 2.653968e+02 2.653968 2.891636e+02 7.340730e+02
min 0.000000e+00 0.000000 0.000000e+00 0.000000e+00
25% 4.670000e+02 4.670000 4.960000e+02 1.212000e+03
50% 6.410000e+02 6.410000 6.830000e+02 1.700000e+03
75% 8.030000e+02 8.030000 8.600000e+02 2.192000e+03
max 1.205000e+03 12.050000 1.323000e+03 3.215000e+03
sum 2.377461e+06 NaN 2.546377e+06 6.315873e+06
expected_count 4.500000e+02 450.000000 4.500000e+02 4.500000e+02

Show Total Tile Counts¶

For each batch, cell line, replicate and markerTotal number of tiles

In [35]:
to_heatmap = total_sum.rename(columns={'n_valid_tiles':'index'})
plot_filtering_heatmap(to_heatmap, extra_index='marker', vmin=None, vmax=None,
                      xlabel = 'Total number of tiles', show_sum=True, figsize=(6,8))

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

Show Total Whole Cell Counts¶

For each batch, cell line, replicate and markerTotal number of tiles

In [36]:
to_heatmap = total_sum.rename(columns={'site_whole_cells_counts_sum':'index'})
plot_filtering_heatmap(to_heatmap, extra_index='marker', vmin=None, vmax=None,
                      xlabel = 'Total number of whole cells', show_sum=True, figsize=(6,8))

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)
/home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/tools/preprocessing_tools/qc_reports/qc_utils.py:381: UserWarning: set_ticklabels() should only be used with a fixed number of ticks, i.e. after set_ticks() or using a FixedLocator.
  ax.set_yticklabels(ax.get_yticklabels(), fontsize=6)

Show Cell Count Statistics per Batch¶

In [37]:
df_no_empty_sites = df_dapi[df_dapi.n_valid_tiles !=0]
plot_cell_count(df_no_empty_sites, lines_order, custom_palette, y='site_cell_count_sum', 
                title='Cell Count Average per Site (from tiles)')

plot_cell_count(df_no_empty_sites, lines_order, custom_palette, y='site_whole_cells_counts_sum',
                title='Whole Cell Count Average per Site')

plot_cell_count(df_no_empty_sites, lines_order, custom_palette, y='site_cell_count',
               title='Cellpose Cell Count Average per Site')

Show Tiles per Site Statistics¶

In [38]:
df_dapi.groupby(['cell_line_cond']).n_valid_tiles.mean()
Out[38]:
cell_line_cond
FUSHeterozygous    3.988985
FUSHomozygous      7.550857
FUSRevertant       6.347472
OPTN               6.315903
SNCA               3.520192
TBK1               8.444027
TDP43              5.720344
WT Untreated       7.866774
WT stress          7.488510
Name: n_valid_tiles, dtype: float64
In [40]:
plot_catplot(df_dapi, custom_palette,reps, x='n_valid_tiles', x_title='valid tiles count', batch_min=4, batch_max=9)

Show Mean of cell count in valid tiles¶

In [41]:
plot_hm(df_dapi, split_by='rep', rows='cell_line', columns='panel', vmax=3)
In [42]:
df_dapi[['cells_count_in_valid_tiles_mean']].mean()
Out[42]:
cells_count_in_valid_tiles_mean    1.503016
dtype: float64
In [43]:
df_dapi[['site_cell_count']].mean()
Out[43]:
site_cell_count    16.915984
dtype: float64
In [44]:
## Only batches 7&8
plot_hm_combine_batches(df_dapi,  batches=['batch7','batch8'], 
                        reps = ['rep1','rep2'],rows='cell_line', columns='panel', vmax=3)

Assessing Staining Reproducibility and Outliers¶

In [52]:
for batch in batches:
    print(batch)
    #batch_num = batch.replace('batch',"")
    run_calc_hist_new(batch,cell_lines_for_disp, markers, root_directory_raw, root_directory_proc,
                           hist_sample=10,sample_size_per_markers=200, ncols=7, nrows=4)
    print("="*30)

batch4

==============================
batch5

==============================
batch6

==============================
batch7

==============================
batch8

==============================
batch9

==============================
In [ ]:
# save notebook as HTML ( the HTML will be saved in the same folder the original script is)
display(Javascript('IPython.notebook.save_checkpoint();'))
os.system(f'jupyter nbconvert --to html tools/preprocessing_tools/qc_reports/qc_report_d8.ipynb --output {NOVA_HOME}/manuscript/preprocessing_qc_reports/qc_report_d8.html')

[NbConvertApp] Converting notebook tools/preprocessing_tools/qc_reports/qc_report_d8.ipynb to html
[NbConvertApp] Writing 24085259 bytes to /home/labs/hornsteinlab/Collaboration/MOmaps_Noam/MOmaps/manuscript/preprocessing_qc_reports/qc_report_d8.html
Out[ ]:
0
In [ ]: